1. CNN
2. Encoder-Decoder
3. Attention
4. Transformers

• Convolutional Neural Networks, or Convolutional Networks, or CNNs, or ConvNets
• For processing data with a grid-like or array topology
  • 1-D grid: time-series data, sensor signal data
  • 2-D grid: image data
  • 3-D grid: video data
• CNNs include four key ideas related to natural signals:
  • Local connections
  • Shared weights
  • Pooling
  • Use of many layers

• Intuition: Neural network with specialized connectivity structure
  • Stacking multiple layers of feature extractors, low-level layers extract local features, and high-level layers extract learn global patterns.
• There are a few distinct types of layers:
  • Convolutional Layer: detecting local features through filters (discrete convolution)
  • Non-linear Layer: normalization via Rectified Linear Unit (ReLU)
  • Pooling Layer: merging similar features

• The core layer of CNNs
• Convolutional layer consists of a set of filters, \(W_{kl}\)
• Each filter covers a spatially small portion of the input data, \(Z_{i,j}\)
• Each filter is convolved across the dimensions of the input data, producing a multidimensional feature map.
• As we convolve the filter, we are computing the dot product between the parameters of the filter and the input.
• Deep Learning algorithm: During training, the network corrects errors and filters are learned, e.g., in Keras, by adjusting weights based on Stochastic Gradient Descent, SGD (stochastic approximation of GD using a randomly selected subset of the data).
• The key architectural characteristics of the convolutional layer is local connectivity and shared weights.

• Intuition: Increase the nonlinearity of the entire architecture without affecting the receptive fields of the convolution layer
• A layer of neurons that applies the non-linear activation function, such as,
  • \(f(x)=max⁡(0,x)\) - Rectified Linear Unit (ReLU);
  fast and most widely used in CNN
  • \(f(x)=\text{tanh}x\)
  • \(f(x)=|\text{tanh}⁡𝑥|\)
  • \(f(x)=(1+𝑒^{−𝑥})^{−1}\) - sigmoid

• Intuition: to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting
• Pooling partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value of the features in that region.

• The convolution, non-linear, and pooling layers are typically used as a set. Multiple sets of the above three layers can appear in a CNN design.
  • Input → Conv. → Non-linear → Pooling → Conv. → Non-linear → Pooling → … → Output
• Recent CNN architectures have 10-20 such layers.
• After a few sets, the output is typically sent to one or two fully connected layers.
  • A fully connected layer is a ordinary neural network layer as in other neural networks.
  • Typical activation function is the sigmoid function.
  • Output is typically class (classification) or real number (regression).

• The final layer of a CNN is determined by the research task.
• Classification: Softmax Layer \[P(y=j|\boldsymbol{x}) = \frac{e^{w_j \cdot x}}{\sum_{k=1}^K{e^{w_k \cdot x}}}\]
  • The outputs are the probabilities of belonging to each class.
• Regression: Linear Layer \[f(\boldsymbol{x}) = \boldsymbol{w} \cdot \boldsymbol{x}\]
  • The output is a real number.

Main CNN idea for text:

Compute vectors for n-grams and group them afterwards

Example: “this takes too long” compute vectors for:

This takes, takes too, too long, this takes too, takes too long, this takes too long

• Prerequisites:
  • Python 3.5+ (https://www.python.org/)
  • TensorFlow (https://www.tensorflow.org/)
  • Keras (https://keras.io/)
    • Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.
• Recommended:
  • NumPy
  • Scikit-Learn
  • NLTK
  • SciPy

• RNN: input sequence is transformed into output sequence in a one-to-one fashion.
• Goal: Develop an architecture capable of generating contextually appropriate, arbitrary length, output sequences
• Applications:
  • Machine translation,
  • Summarization,
  • Question answering,
  • Dialogue modeling.

Most significant change: new set of weights, U - connect the hidden layer from the previous time step to the current hidden layer. - determine how the network should make use of past context in calculating the output for the current input.

• Trained Neural Language Model can be used to generate novel sequences
• Or to complete a given sequence (until end of sentence token <> is generated)

• Translation as Sentence Completion!

• Encoder generates a contextualized representation of the input (last state).
• Decoder takes that state and autoregressively generates a sequence of outputs

Abstracting away from these choices

1. Encoder: accepts an input sequence, \(x_{1:n}\) and generates a corresponding sequence of contextualized representations, \(h_{1:n}\)
2. Context vector \(c\): function of \(h_{1:n}\) and conveys the essence of the input to the decoder.
3. Decoder: accepts \(c\) as input and generates an arbitrary length sequence of hidden states \(h_{1:m}\) from which a corresponding sequence of output states \(y_{1:m}\) can be obtained.

Widely used encoder design: stacked Bi-LSTMs - Contextualized representations for each time step: hidden states from top layers from the forward and backward passes

• produce an output sequence an element at a time

• Sample soft-max distribution (OK for generating novel output, not OK for e.g. MT or Summ)
• Most likely output (doesn’t guarantee individual choices being made make sense together)

Context vector \(c\): function of \(h_{1:n}\) and conveys the essence of the input to the decoder.

Context vector \(c\): function of \(h_{1:n}\) and conveys the essence of the input to the decoder.

Flexible?
- Different for each \(h_i\) - Flexibly combining the \(h_j\)

• Replace static context vector with dynamic \(c_i\)
• derived from the encoder hidden states at each point \(i\) during decoding

Ideas: - should be a linear combination of those states \[c_i = \sum_j{\alpha_{ij}h^e_j}\] - \(\alpha_{ij}\) should depend on?

• Compute a vector of scores that capture the relevance of each encoder hidden state to the decoder state \(h_{i-1}^d\) \[score(h_{i-1}^d, h_j^e)\]

• Just the similarity \[score(h_{i-1}^d, h_j^e) = h_{i-1}^d \cdot h_j^e\]

• Give network the ability to learn which aspects of similarity between the decoder and encoder states are important to the current application.

\[score(h_{i-1}^d, h_j^e) = h_{i-1}^d W_S h_j^e\]

• Create vector of weights by normalizing scores

\[ \begin{align} a_{ij} &= \text{softmax}(score(h_{i-1}^d, h_j^e)\ \forall j \in e) \\ &= \frac{exp(score(h_{i-1}^d, h_j^e))}{\sum_k{exp(score(h_{i-1}^d, h_k^e))}} \end{align} \]

• Goal achieved: compute a fixed-length context vector for the current decoder state by taking a weighted average over all the encoder hidden states.

• Just an introduction: These are two valuable resources to learn more details on the architecture and implementation

• http://nlp.seas.harvard.edu/2018/04/03/attention.html

• https://jalammar.github.io/illustrated-transformer/ (slides come from this source)

• Will only look at the ENCODER(s) part in detail

Key property of Transformer: word in each position flows through its own path in the encoder. - There are dependencies between these paths in the self-attention layer. - Feed-forward layer does not have those dependencies => various paths can be executed in parallel !

• dependencies in self-attention layer.
• No dependencies in Feed-forward layer

While processing each word it allows to look at other positions in the input sequence for clues to build a better encoding for this word.

Step1: create three vectors from each of the encoder’s input vectors:

Query, a Key, Value (typically smaller dimension).

by multiplying the embedding by three matrices that we trained during the training process.

Step 2: calculate a score (like we have seen for regular attention!)  how much focus to place on other parts of the input sentence as we encode a word at a certain position.

Take dot product of the query vector with the key vector of the respective word we’re scoring.

E.g., Processing the self-attention for word “Thinking” in position \(\text{#}1\), the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2.

• Step 3 divide scores by the square root of the dimension of the key vectors (more stable gradients).
• Step 4 pass result through a softmax operation. (all positive and add up to 1)

Intuition: softmax score determines how much each word will be expressed at this position.

Step6 : sum up the weighted value vectors. This produces the output of the self-attention layer at this position

Step6 : sum up the weighted value vectors. This produces the output of the self-attention layer at this position

More details: - What we have seen for a word is done for all words (using matrices) - Need to encode position of words - And improved using a mechanism called “multi-headed” attention

(kind of like multiple filters for CNN)

see https://jalammar.github.io/illustrated-transformer/

• Relies on most of the concepts on the encoder side
• See animation on https://jalammar.github.io/illustrated-transformer/